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ABSTRACT 

Learning analytics can promote pedagogically informed use of learner data, which can steer the progress of technology 
mediated learning across several learning contexts. This paper presents the application of analytics to a mobile learning 
solution and demonstrates how a pedagogical sense was inferred from the data. Further, this inference was used to 
identify undesirable learning behavior and estimate the effectiveness of the learning strategy. 
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1. INTRODUCTION 

Learning Analytics is an evolving field that holds the promise of augmenting teaching and learning 
experiences through its data-driven approach, and often traces its roots in related domains like educational 
data mining, web analytics, semantic analysis and so on (Ferguson, 2012). Bienkowski et al., (2012) 
postulates that learning analytics deals with the interpretation of learner data to be able to assess the learner 
preferences, detect the progress along with embedded problems and predict the future performance. Thus, it 
involves data collection, measurement, analysis, reporting, and representation of mined data in a meaningful 
way such that both learners and trainers can make sense of the current state of learning and optimize the 
learning interventions (Bichsel, 2012). Alongside of this, researchers proclaim that analytics maximize the 
use of data by converting information into insight, thus supporting in strategic decision making, which can 
impact the overarching goal of increased learner engagement and enhancement (Macfadyen and Dawson 
(2012), Cooper (2012)). 

With the increasing adoption of technology mediated learning, there is a higher scope to trace the learner 
trails and gather the data pertaining to learner, context, outcome, resources, actions and more (Verbert et al., 
2012). In this pretext, Siemens and Gasevic (2012) note the value proposition drawn from the online data 
trails, which reflects learner engagement in social networks, fine grained learner behaviors in an online or 
mobile learning environment. This affordance of collecting learner traces to improve learning engagements is 
stronger and realistic in mobile medium than a computer-based learning environment. That is because mobile 
devices are ubiquitous, sensor enabled and always turned on, which attribute to an intensive tracking 
mechanism resulting in large and realistic data sets (Duval, 2012). Additionally, due to their characteristic 
ubiquitous nature, mobile devices augment the job performance effectively. In support of this notion, Udell 
(2012) mentions that performance support tools are accessible anywhere and anytime, enabling the 
employees work better in their jobs and hence emphasizes that mobile medium is a perfect channel to support 
the performance. 

Despite a radically transforming ability towards pedagogy, there has been minimal evidence in 
identifying the right set of questions to mine the data or understand what data is required to arrive at certain 
thematic conclusions. In other words, the on-going research hasn’t yet revealed an established method or 
methodology for learning analytics to inform the teaching and learning design (Dawson et al., 2008). 
Relating this idea to mobile learning, Aljohani and Davis (2012) highlight the limited research conducted 
about using mobile learners’ data in sense making. 
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In the light of this research gap, this paper addresses the deficit by presenting a Mobile Learning 
Analytics case study. The paper introduces a mobile learning solution that aims to support on-the-job 
performance and demonstrates how analytics has helped identify trends and patterns, from mobile learner 
driven data. This analysis has helped accomplish one of the key objectives of Learning and Knowledge 
Analytics that is to spot the undesirable learning behavior (Verbert et al., 2012), which has resulted in 
detecting learners’ performance gaps and estimating the effectiveness of the mobile learning app. 


2. THE FRAMEWORK 

The study begins by laying out a bottom-up framework for characterizing the nature of metrics available to 
mobile learning designers, and then recommends a possible strategy for future applications. For the sake of 
simplicity, we refer to a smartphone app (“app”) as a prototypical mobile learning solution in the following 
narrative. The primary classification derived from the boundaries of the learner’s interaction with an app 
yields two types of metrics: intrinsic and extrinsic. Intrinsic metrics pertain to the information collected from 
the point the learners launch the app to the point they close it. Extrinsic metrics pertain to the information that 
can relay insight into learners’ ratings post-use of the app, what the learners were performing immediately 
before launching the app (hence triggering the need to seek learning), as well as what they intend to perform 
immediately after closing the app (which, in an ideal case, would be to apply or to employ the designed 
learning objective from their interaction in the app). 

Furthermore, intrinsic metrics can be broken down in to subject-based, time-based and location-based 
metrics. Subject-based intrinsic metrics include data points that capture the identity of the learners and what 
they were interacting with in the app. Time-based metrics include data points that capture the time, duration, 
frequency and recentness of interactions, along with the span of idle-times. Lastly, Location-based metrics 
include data points that capture the geo-location, proximity to points of interest as well as the rate of change 
of learner’s location. The points of interest can be deduced from locations that may lie on the learner’s 
physical journey leading up to the actual engagement in the designed learning. For instance, if an app is 
designed to provide European competitor fact-sheets to a sales person, who is based in Chicago, IL (USA), 
and is traveling to France for a conference, then some key points of interest may be the home address, in- 
transit airports, conference location, and so on. 

2.1 Drawing Analytical Insight from Intrinsic and Extrinsic Metrics 

The classification above only gives us data points that can be captured using technology that is available to 
mobile devices via various programming interfaces. However, to draw analytical insight, it is essential to 
understand who interacts with the app and what the underlying context is. This resonates with the idea of 
Learner to Learning Material interaction, which is Learner to App in our context, that helps realize mobile 
learning analytics (Aljohani and Davis, 2012). This enables analysts to evaluate whether the level, frequency 
and nature of interaction result in designated learning objective. 

The mobile device, be it a tablet or a smartphone, has a multitude of factors that define the context of the 
learner (such as battery and network strengths, social interruptions, distractions, to name a few), much 
beyond the scope of content and learning design. The possible combinations of the intrinsic and extrinsic 
metrics that allow us to gain real insight into the efficacy of a design are numerous, and often overwhelming.. 

As the following case study shall depict, the iterative method of analyzing metrics and drawing insights 
can assist in identifying the weak elements of content design that lowered the learner engagement along with 
revealing the gap in learner-app interaction strategy. The intrinsic metrics that the case study captured 
included item-level responses alongside the time and location markers of each interaction. We chose to not 
include any extrinsic metrics in the initial release of the app, to allow us to focus on how content and 
instructional design factored into the learning design. 

2.2 Case Study 

In 2010, we designed an iPad app that allowed managers to collect customer needs by answering a certain set 
of questions. The intention was that managers would launch the application, create a new “customer session” 
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or open an existing one. Next, they would see a set of multiple choice, text entry, and slider questions that 
would guide them in filling out a mobile form. Initial conversations and requirements gathering from 
potential users of the app allowed us to incorporate key features that accommodated the context of the end- 
user. For instance, the managers identified that the app may be used in locations that may have spotty 
network connectivity, which required us to feature a sync-capability to record data local to the device and 
then merging with a central record store. Another feature accommodating the manager’s contextual needs 
included a shared analysis dashboard that allowed comparing client results with other managers of the same 
organization. As for the content, the order and nature of the questions were a linear, scrollable list of 
questions in a predetermined order, based on expert advice and recommendations. 

The learning objective was to help the manager gather the customer requirements in a holistic manner, so 
that they can relay the information down to development teams as well as use it in future 
interactions/engagements. In this process, the metrics about user behavior as well app-states (intrinsic, in 
nature) were tracked to identify the gaps between what was expected (as outlined by their needs) and what 
was observed (actual context-defined behaviors). To track these metrics, we created a native app (written in 
the Obj-C programming language) that interacted via an event and value analytics interface with a custom 
Google Analytics instance. The metrics collected were analyzed using a combination of the reporting 
available from Google Analytics as well as Microsoft Excel data tools. Table 1 depicts the intent and detailed 
metrics, as per the first rollout of the app. 


Table 1. Intrinsic metrics captured in the initial rollout of the app 


Interaction 

Observation Intent 

Metrics 

Questions 
Dashboard Access 
Session Interaction 

Design effectiveness 
Design and behavior 
Contextual user behavior 

Duration, Time, Frequency 

Order of access, Time, Frequency (per session) 

Duration, Time, Number of Questions completed 


Based on these metrics, few vital performance problems were detected that resulted due to deviation from 
the expected learner behavior in using the mobile app. Given the limited scope, this paper introduces 3 
different scenarios that were analyzed in the context of metrics and presents the insights drawn along with the 
indicative corrective measures. The data revealed some key findings. 

First, we expected that the customer “sessions” would depict a high level of completion of the survey 
(more than 90% of questions answered). Instead, the customer “sessions” were often started during business 
hours, left incomplete (more than 20% questions unanswered) by the managers, while averaging the same 
total duration with very little variance. This most likely indicated that the managers were using the app 
during conversations with their customers, while following the question content, but the structure and order 
of the questions may not have assisted in actual data entry. These metrics were used to iterate and refine the 
design of the questionnaire. 

Second, we didn’t expect to see any patterns in what type of questions were answered. However, our 
metrics revealed that the slider type of questions was consistently left unanswered. Additionally, the slider 
questions that were answered had an unusually prolonged duration of interaction (user sliding the slider knob 
for a long time). This indicated that the design of the actual interaction (sliding knob of slider) demanded us 
to look at alternatives. Could it be that the manager wasn’t able to use the slider question effectively on the 
iPad? The iteration around this metric was to alter the actual design of the slider question in an aim to drive a 
higher answer count and considerably less interaction time. The alternate design that helped us meet these 
revised expectations is depicted in Figure 1 . 


slider wiih no helper Lcxi (before) 



Figure 1. Altering design of the slider question type to allow hints 
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Third, the manager most frequently switched between the customer sessions and the dashboard, rather 
than spending prolonged times between accessing the two views. In this case, we could draw insight that the 
learner behavior within the context of recording customer responses involved going back and forth between 
the questions and the summary dashboard. This potentially required us to redesign how the dashboard was 
not a separate element, but instead needed to provide a level of performance support to the survey 
functionality of the questions. The revised dashboard was accessible as a popup from the questions, rather 
than a distracting view separated from the question interface. This is an example of a metric that allowed us 
to draw insight into how the manager was using the app as a performance support tool during a 
conversational context. 


3. CONCLUSION 

As the case study has depicted, the iterative method of analyzing metrics and drawing insights can support in 
(a) comprehensively identifying the attributes of the content design that may result in low engagement of the 
mobile learner; and, (b) falsifying assumptions around mobile design that may have originally led to less 
effective learning solutions. Thus, analytics has enabled an early detection of undesired behavior in job 
performance and as well provided a “before and after” snapshot of learner interactions after changing the 
learning activity design. With increased metrics, this analysis can be taken to the next level of monitoring the 
learner behavior, to identify the precise points of disconnection for every single learner and perhaps adapt the 
design and complexity of questions accordingly. Based on this, we propose an iterative 3-step approach - 
hypothesizing, tabulating and abstracting towards using analytics in mobile learning. 

Let’s consider an iPad application that provides help content to culinary students. One hypothesis may be 
to question the effectiveness of the 10-minute videos that are used in the application. Next, we tabulate 
metrics that may relate to students’ engagement with the video content. For example, let’s consider the 
learners’ durations for and frequencies of watching each video, the original length and the targeted objective 
of each video, and the times between consecutive accesses of each video. Lastly, we shift the level of 
abstraction so we can group items to observe learner trends relating to the nature of the video shown (e.g. 
videos about cooking v/s videos about preparation). If we find that a certain video category (e.g. preparation 
videos) is least watched, both in frequency and duration, the efficacy of video as a means of conveying that 
information may not be most efficient. At this point, we iterate our analysis by reframing a new hypothesis 
around the category of videos that reveal this behavior (i.e. preparation videos in our example). New 
tabulations may include metrics like learner ratings. However, limiting analytics to learner ratings may only 
capture learner perception of value, rather than how well the learning design is aligned to the learner’s 
context. 

Drawing on the approach, this paper concludes by posing an evolving framework for mobile learning 
analytics which observes mediation between a) learner interactions (learner-device and learner-material) and 
b) intrinsic and extrinsic metrics, which undergoes an iterative analysis to translate information into insight 
(Figure 2). 
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